Skip to content

test(integration): add opt-in parallel test execution - #252

Draft
cleithner-comcast wants to merge 10 commits into
mainfrom
cleith/dev/testSpeedup
Draft

test(integration): add opt-in parallel test execution#252
cleithner-comcast wants to merge 10 commits into
mainfrom
cleith/dev/testSpeedup

Conversation

@cleithner-comcast

Copy link
Copy Markdown
Contributor

Run the integration suite in parallel via 'py_test.sh --parallel[=N]' (serial by default so individual/interactive runs keep readable, interleaved logs). CI opts in via run_integration_tests.sh. Default workers is min(nproc/4, 64) -- roughly half the physical cores, since each worker drives a multi-threaded Barton plus a matter.js node process and nproc counts hyperthreads.

Supporting changes for concurrent test processes:

  • matter: bind ephemeral operational/UDC ports under BARTON_CONFIG_MATTER_USE_RANDOM_PORT so concurrent Barton servers don't collide on CHIP_PORT/CHIP_UDC_PORT (5540/5550).
  • matter: barton patch 0003 raises the CHIP commissioner's discovered-node cache 10 -> 128. On the shared mDNS plane every commissioner sees every device's advertisement, so concurrent commissionings overflowed the 10-slot cache ('Insufficient space') and discovery timed out -- capping reliable parallelism at ~4. With 128 no overflow occurs even at 64 workers.
  • test env: per-process storage on tmpfs with cleanup on process exit regardless of outcome; sweep the shared compile-time ~/.brtn-ds once at session end.
  • pin the fixed-port zhal mock tests (18443/8711) to one xdist worker via xdist_group + loadgroup; strip xdist's @group nodeid suffix in the subprocess-per-test runner.
  • raise commission/ready/resource timeouts (5->30, 10->30) to tolerate load.
  • py_test.sh: resolve repo root from script location so PYTHONPATH and LD_LIBRARY_PATH point at this worktree, not a stale primary clone.

@cleithner-comcast cleithner-comcast left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition to the below, need to look at how the suite dumps logs when an error occurs. We need sane separators if logs are interleaved so that they can be parsed.

Comment thread docker/version Outdated
Comment thread testing/environment/base_environment_orchestrator.py Outdated
Comment thread testing/conftest.py Outdated
Comment thread testing/py_test.sh
Run the integration suite in parallel via 'py_test.sh --parallel[=N]' (serial by default so individual/interactive runs keep readable, interleaved logs). CI opts in via run_integration_tests.sh. Default workers is min(nproc/4, 64) -- roughly half the physical cores, since each worker drives a multi-threaded Barton plus a matter.js node process and nproc counts hyperthreads.

Supporting changes for concurrent test processes:
- matter: bind ephemeral operational/UDC ports under BARTON_CONFIG_MATTER_USE_RANDOM_PORT so concurrent Barton servers don't collide on CHIP_PORT/CHIP_UDC_PORT (5540/5550).
- matter: barton patch 0003 raises the CHIP commissioner's discovered-node cache 10 -> 128. On the shared mDNS plane every commissioner sees every device's advertisement, so concurrent commissionings overflowed the 10-slot cache ('Insufficient space') and discovery timed out -- capping reliable parallelism at ~4. With 128 no overflow occurs even at 64 workers.
- test env: per-process storage on tmpfs with cleanup on process exit regardless of outcome; sweep the shared compile-time ~/.brtn-ds once at session end.
- pin the fixed-port zhal mock tests (18443/8711) to one xdist worker via xdist_group + loadgroup; strip xdist's @group nodeid suffix in the subprocess-per-test runner.
- raise commission/ready/resource timeouts (5->30, 10->30) to tolerate load.
- py_test.sh: resolve repo root from script location so PYTHONPATH and LD_LIBRARY_PATH point at this worktree, not a stale primary clone.
The integration suite can now run in parallel via 'py_test.sh --parallel' (pytest-xdist). Install pytest-xdist in the builder image and bump the builder version 2.13 -> 2.14 so a fresh container and CI have the dependency.
…r sweep

Per review feedback:
- Restore the commission/ready/resource timeouts to their original values (5/5/10); the increases masked failures and should not have changed.
- Remove the pytest_sessionfinish sweep of ~/.brtn-ds. That compile-time CHIP config dir may be shared with reference-app runs, so the test suite must not delete it; it is simply reused instead.
2.15 is reserved for the webrtc work; this parallel-test image needs its own version so the pytest-xdist install is picked up.
Under --parallel several failing tests' captured logs are reported together (and background C-library writes to fd 1/2 can interleave). Wrap each subprocess's dumped output in greppable 'BARTON TEST OUTPUT BEGIN/END [<nodeid>]' banners so the failure dump stays attributable and parseable.
Add --client-ready-timeout / --device-added-timeout / --resource-value-timeout pytest options (defaults 5/5/30), resolved into testing/utils/timeouts.py in pytest_configure and forwarded into each per-test subprocess so the child running the test sees the same values. The wait helpers (wait_for_client_to_be_ready, wait_for_device_added, wait_for_resource_value) read them at call time.

py_test.sh --parallel raises them to 30/30/30 so concurrent commissioning has headroom, while serial/interactive runs keep the fast-failing defaults for quick feedback.
Under parallel test execution, commissioning intermittently failed because
the harness commissioned via the manual pairing code, which carries only a
4-bit short discriminator. CHIP's SetUpCodePairer then discovers with a
kShortDiscriminator filter, matching any device sharing that short
discriminator. With many devices advertising concurrently (only 16 short
buckets), the commissioner frequently matched the wrong device and PASE'd to
its port, which never completes -> PASE/discovery timeout.

Add generate_qr_code() (full 12-bit discriminator, Base38-encoded MT: payload)
alongside the existing manual pairing code logic, and commission via the QR
code so the commissioner matches the exact device. Barton's ParseSetupPayload
auto-detects the MT: prefix and uses the long-discriminator parser.

At 64 workers the full suite goes from 15 failed / 47 passed to 62 passed.
The former nproc/4 divisor was extra headroom against commissioning timeouts
that were actually the short-discriminator collision (now fixed by commissioning
via the full-discriminator QR code), not CPU starvation. With that gone, one
worker per physical core (nproc/2, since nproc counts hyperthreads) is reliable.

Pin CI to --parallel=3: the runners have 4 cores, and a 4-core sweep showed 3
workers is ~30% faster than 2 (70s vs 100s) with zero flakes across repeated
runs, while keeping a core free for the OS/dbus/otbr. The nproc/2 default would
otherwise resolve to 2 on those runners.
# Conflicts:
#	docker/version
#	testing/environment/base_environment_orchestrator.py
Squashed follow-ups to the origin/main merge:

- Pin GI_TYPELIB_PATH to the worktree's build/core so parallel/worktree runs
  load this tree's freshly-built BCore typelib. main bumped BCore to major
  version 4, and the container's GI_TYPELIB_PATH points at the provisioning
  clone, so without this a worktree resolves the wrong typelib (or fails with
  'Namespace BCore not available').

- Make the side-band channel address- and lifecycle-correct: connect the client
  to 127.0.0.1 to match the server's bind (instead of dual-stack 'localhost'),
  and have the server close each connection after responding, so no kept-alive
  sockets linger on the loopback control channel under load.

- Default to min(nproc/4, 32) workers, about one worker per two physical cores.
  Commissioning is a crypto-heavy PASE/CASE burst that runs concurrently on both
  the commissioner (Barton) and the target (a matter.js node process), so it
  needs ~2 physical cores while it runs; packing one worker per physical core
  starves the target and it misses the PASE handshake. Starvation is fixed with
  resources, not retries or inflated timeouts. Validated 0 flakes over 50 runs.

- Specify the CI worker count in the workflow (--parallel=3) rather than in
  run_integration_tests.sh, since the runner's core count (GitHub-hosted runners
  have 4) is a property of the CI environment. The runner script now forwards
  its arguments to py_test.sh.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant